1. Field of the Invention
The present invention relates to a semiconductor neural network device which electronically implements a neural network, and more particularly, it relates to a synapse expressing circuit which expresses a synapse for coupling neurons with prescribed connection strength. More specifically, the present invention relates to a synapse expressing circuit which can set a synapse load at an arbitrary value, and a semiconductor neural network device which includes such synapse expressing circuits.
2. Description of the Background Art
Among various computing techniques which are modelled on vital cells (neurons), there is a parallel information processing machine called a Boltzmann machine. The Boltzmann machine, proposed by D. H. Ackley et al. in 1985, is a kind of a mathematical model expressing a neural network, which is characterized in stochastic state transition of neurons. When connection strength (hereinafter referred to as a synapse load) Wij between neurons i and j is symmetrical (Wij=Wji) with no self coupling (Wii=0), stationary distribution p(z) of the system (neural network) is provided by the following Boltzmann distribution, assuming that a parameter T expressing the temperature of the system has a finite value: EQU p(z)=C.multidot.exp(exp(-U(z)/T)
where U(z) represents the potential function of the system, z represents the state in the system taken by the neurons, and C represents a normalization coefficient.
With introduction of the aforementioned probability as to state determination of the neurons, it is expected that the neural network system is not trapped by a local minima of state energy but converged to the global minima. In other words, it is expected that a most likely solution can be found. The Boltzmann machine, which is suitable for solving a complicated cost problem, and non-algorithmic problems such as pattern recognition and associative memory, is in contrast with a Turing machine, which requires algorithms for solving problems.
For an apparatus which can efficiently express such a Boltzmann machine at a high speed, some attempts have already been made to implement a strong parallel processing system which imitates the operation of a neutral network by a semiconductor integrated circuit. Before discussing the structure and operation of such a conventional integrated semiconductor neural network device, the operation principle of the Boltzmann machine is now briefly described.
FIG. 10 illustrates the structure and the operation principle of a general neuron model. Referring to FIG. 10, a neuron unit i includes an input part A which receives output signals Sk, Sj and Sm from other units k, j, and m, a conversion part B which converts signals received from the input part A in accordance with a predetermined rule, and an output part C which outputs a signal received from the conversion part.
The input part A has prescribed synapse loads W (synapse loads are hereafter generically denoted by symbol W) with respect to the neuron units k, i and m. For example, the output signal Sk from the neuron unit k is converted to Wik.multidot.Sk with a synapse load Wik in the input. part A, and transmitted to the conversion part B.
The conversion part B is fired when the total sum of the signals received from the input part A satisfies a certain condition, and transmits a signal to the output part C. When a neuron unit is related to a vital cell, the input part A corresponds to a dendrite and the conversion part B corresponds to the body of a nerve cell, while the output part C corresponds to an axon.
In this neuron model, it is assumed that each neuron unit takes two states of Si-0 (non-firing state) and Si= 1 (firing state). Each neuron unit updates its state in response to inputs received therein. The total input of the neuron unit i is defined as follows: EQU Ui=.SIGMA.Wij.multidot.Sj+Wii
This total sum .SIGMA. is obtained for j. Symmetrical synapse coupling of Wij=Wji is assumed here, while -Wii corresponds to the threshold value of the neuron unit i.
The states of neuron units are stochastically updated asynchronously with each other. When the neuron unit i updates its state, the new state is 1 in the following probability p(Si=1): EQU p(Si=1)=1/(1=exp(-Ui/T))
where T represents a parameter, serving similarly to the temperature in a physical system, which takes a positive value. This parameter is generally called "temperature".
FIG. 11 illustrates the relation between the total sum Ui of the inputs and the probability p(Si=1) as to respective temperature levels T. As understood from FIG. 11, the neuron unit i takes either "0" or "1" with a probability of 1/2 substantially at random when the temperature T is high. When the temperature T approximates zero, on the other hand, the neuron unit i substantially decision-theoretically follows such a threshold logic that the state is "1" when the total sum of the inputs exceeds a certain threshold value.
The state of the Boltzmann machine at a certain time is expressed by combination of ONs (S=1) and OFFs (S=0) of all units, where S generically represents output signals of the neuron units. For a system which is in a certain state, energy E is defined as follows: EQU E=-.SIGMA.Wij.multidot.Si.multidot.Sj
The total sum .SIGMA. is obtained as to all subscripts i and j satisfying a relation of i&lt;j. In the above relation, it is assumed that the threshold value of each neuron unit is zero. Such a state is implemented by providing a unit which is normally in an ON state (S=1) to each neuron unit, and whose connection strength is equal to and inverse in sign to the threshold value of an associated unit.
When the respective neuron units starting at arbitrary initial states continue operations, the Boltzmann machine approaches a stochastic equilibrium state which is determined by the synapse loads W of the respective neuron units. As hereinabove described, the Boltzmann machine takes a state .alpha. in the following probability P(.alpha.): EQU P(.alpha.)=C.multidot.exp(-E.alpha./T)
where E.alpha. represents energy of the neural network system in the state .alpha..
In such a Boltzmann machine, a technique called simulated annealing is employed in order to reach the global energy minima. A relative probability of two global states .alpha. and .beta. is expressed as follows: EQU P(.alpha.)/P(.beta.)=exp(-(E.alpha.-E.beta.)/T)
The minimum energy state takes place in the highest probability at an arbitrary temperature. In general, it takes a long time to reach a thermal equilibrium state, and hence it is regarded preferable to start annealing from a high temperature level to gradually reduce the temperature. In general, this state transition is similar to such a state that each crystal atom in a crystal lattice is transferred to a position providing the minimum energy state at a given temperature.
In the Boltzmann machine, it is necessary to find out such weight i.e., synapse loads W that the network itself can implement probability distribution of input/output data as correct as possible with no external provision of such distribution. As a basic equation of a learning rule employed for adjusting such synapse loads W, the following equation is often used: EQU .DELTA.Wij=.eta..multidot.(p.sup.+ ij-P.sup.- ij) (1)
where p.sup.+ ij represents such an expected value that the states of both neuron units i and j are "1" when the neural network is driven by externally supplied educator information to reach an equilibrium state, and p.sup.- ij represents such an expected value that the states of both neuron units i and j are "1" when no educator information is externally supplied. In the above equation (1), the term of p.sup.+ ij indicates that connection between the neuron units i and j is reinforced when both of the same are activated. This corresponds to a reinforced learning mechanism of synapse coupling called Hebb's learning rule.
On the other hand, the term of p.sup.- ij indicates that connection (synapse load Wij) between the neuron units i and j is weakened (reduced) when both of these units are activated with no external output supply. This is called anti-learning in general. A learning algorithm in the Boltzmann machine is now briefly described.
The learning algorithm in the Boltzmann machine includes an operation 1 (plus (+) phase), an operation 2 (minus (-) phase), and an operation 3.
Operation 1 (Plus Phase): States of input units and output units (visible units) are fixed to specific patterns which are shown by input data and output data (educator data) according to pattern appearance probabilities, respectively. The operation 1 includes (a) an annealing process, (b) a data collection process and (c) a process of obtaining p.sup.+ ij. In the annealing process (a), the states of the respective units are changed for respective temperature levels T along the following equations (2) and (3): EQU .DELTA.Ei=.SIGMA.Wij.multidot.Sj (2) EQU Pi=1/(1+exp(-.DELTA.Ei/T) (3)
In the equation (2), the total sum .SIGMA. is obtained in relation to the subscript j. This equation (2) provides an energy gap between states Si of "0" and "1" of the unit i with respect to the energy E of the overall neural network. The equation (3) provides such a probability that the unit i enters a new state Si of "1" when this energy gap takes place. In the aforementioned process (a), the temperature T, which is started from a high level, is successively reduced. When the temperature T is reduced to a low level and a prescribed annealing procedure is terminated, the neural network is assumed to be relaxed to a relatively low energy state, to reach a thermal equilibrium state.
The data collection process (b) is adapted to obtain such a number of times that respective states S of units which are coupled to each other are "1" after the annealing process (a) is repeated a prescribed number of times.
The process (c) of obtaining p.sup.+ ij is adapted to obtain an average value of data obtained in the process (b) after the annealing process (a) and the data collection process (b) are repeated a prescribed number of times in correspondence to an educator pattern, to assume that the average value is p.sup.+ ij.
The operation 2 (minus phase) similarly includes an annealing process (a), a data collection process (b) and a process (c) of obtaining p.sup.- ij. The processes (a), (b) and (c) in the operation 2 are similar to those in the operation 1 (plus phase). In the operation 2 (minus phase), however, states of only units (input units) associated to input data are fixed according to appearance probabilities of educator data. In the operation 2, the processes (a), (b) and (c) are so repeated that an average value thereafter obtained in the process (c) is assumed to be p.sup.- ij, similarly to the operation 1.
In the operation 3, the synapse load Wij is changed with the as-obtained average values p.sup.+ ij and p.sup.- ij in accordance with the following relational expression: EQU .DELTA.Wij=.eta..multidot.(p.sup.+ ij -p.sup.- ij) (4)
where .eta. represents a positive constant which determines the degree of a single change of the synapse load Wij. As clearly understood from the above expression (4), the amount of change of the synapse load Wij is determined only by the states of the two units i and j which are coupled to each other. The final object of learning is to converge the amount .DELTA.Wij of change which is expressed in the expression (4) to a value as small as possible, ideally to zero.
Various apparatuses have been proposed which implement a neural network (called Boltzmann machine) having the aforementioned learning function, by a semiconductor electronic circuit. The inventor et al. have already proposed a semiconductor neural network integrated circuit device having a structure suitable for integration as well as high-speed operability and high learning efficiency (refer to Japanese Patent Laying-Open No. 3-80379).
FIG. 12 shows an exemplary overall structure of a semiconductor neural network integrated circuit device proposed by the inventor et al. The integrated circuit device shown in FIG. 12 is to implement a neural network having five neurons. Referring to FIG. 12, the neural network integrated circuit device includes a column of five neuron units NU1, NU2, NU3, NU4 and NU5, and synapse expressing circuits SY1 to SY10 which are arranged substantially in the form of a right-angled triangle. Respective input parts of the neuron units NU1 to NU5 are connected with dendrite signal lines DE1, DE2, DE3, DE4 and DE5 respectively. The respective neuron units NU1 to NU5 compare signals on the corresponding dendrite signal lines DE1 to DE5 with annealing information contained therein, to generate state signals S1 to S5 of "1" or "0" on the basis of the results of the comparison.
The neural network integrated circuit device further includes axon signal lines AX1, AX2, AX3, AX4 and AX5 which transmit state signals SI1, SI2, SI3, SI4 and SI5 respectively. The state signals SI1 to SI5 may be externally supplied input data, or state signals transmitted from another layer.
Each of the synapse expressing circuits SY1 to SY10 transmits a signal W.multidot.S obtained by weighting the state signal SI received from the corresponding axon signal line AX (symbol AX generically denotes the axon signal lines AX1 to AX5) with a synapse load W stored therein, to the corresponding dendrite signal line DE (symbol DE generically denotes the dendrite signal lines DE1 to DE5).
In the Boltzmann machine which is the model of this semiconductor neural network integrated circuit device, the synapse load W is symmetrical such that Wij=Wji. Thus, each synapse expressing circuit SY (symbol SY generically denotes the synapse expressing circuits SY1 to SY5) provides two synapse loads.
The axon signal line AX1 is connected to first axon signal input terminals of the synapse expressing circuits SY1, SY2, SY3 and SY4. The axon signal line AX2 is connected to a second axon signal input terminal of the synapse expressing circuit SY1 and respective first axon signal input terminals of the synapse expressing circuits SY5, SY6 and SY7. The axon signal line AX3 is connected to respective second axon signal input terminals of the synapse expressing circuits SY2 and SY5 and respective first axon signal input terminals of the synapse expressing circuits SY8 and SY9. The axon signal line AX4 is connected to second axon signal input terminals of the synapse expressing circuits SY3, SY6 and SY8 and a first axon signal input terminal of the synapse expressing circuit SY10. The axon signal line AX5 is connected to second axon signal input terminals of the synapse expressing circuits SY4, SY7, SY9 and SY10.
The dendrite signal line DE1 adds up output signals received from first dendrite signal output terminals of the synapse expressing circuits SY1, SY2, SY3 and SY4, and transmits the result to the neuron unit NU1. The dendrite signal line DE2 adds up an output signal received from a second dendrite signal output terminal of the synapse expressing circuit SY1 with those received from first dendrite signal output terminals of the synapse expressing circuits SY5, SY6 and SY7, and transmits the result to the neuron unit NU2. The dendrite signal line DE3 adds up output signals received from second dendrite signal output terminals of the synapse expressing circuits SY2 and SY5 with those received from first dendrite signal output terminals of the synapse expressing circuits SY8 and SY9, and transmits the result to the neuron unit NU3.
The dendrite signal line DE4 adds up output signals received from second dendrite signal output terminals of the synapse expressing circuits SY3, SY6 and SY8 with that received from a first dendrite signal output terminal of the synapse expressing circuit SY10, and transmits the result to the neuron unit NU4. The dendrite signal line DE5 adds up output signals received from second dendrite signal output terminals of the synapse expressing circuits SY4, SY7, SY9 and SY10, and transmits the result to the neuron unit NU5. The neuron units NU1 to NU5 enter firing or non-firing states in response to the signals transmitted through the corresponding dendrite signal lines DE1 to DE5.
FIG. 13 is a block diagram schematically illustrating the structure of each synapse expressing circuit SY shown in FIG. 12. Referring to FIG. 13, the synapse expressing circuit SY comprises a synapse load value storage circuit 101 which stores synapse load value information, a learning control circuit 110 which generates a synapse load correction signal in response to associated two state signals (axon signals) Si and Sj in a learning mode of the neural network, a synapse load correction circuit 103 which corrects the synapse load value information stored in the synapse load value storage circuit 101 in response to the synapse load correction signal, a synapse coupling expressing circuit 105 which weights the state signal Sj received at the second axon signal input terminal with the synapse load value stored in the synapse load storage circuit 101 and transmits the weighted signal Wij.multidot.Sj to a dendrite signal line DEj, and another synapse coupling expressing circuit 107 which adds the synapse load value being stored in the synapse load value storage circuit 101 to the state signal Si being transmitted to the first axon signal input terminal to generate a signal Wji.multidot.Si and transmits the same onto another dendrite signal line DEi.
The synapse load expressed by the synapse expressing circuit SY is symmetrical such that Wji=Wij. This synapse load value information is stored in the synapse load value storage circuit 101. The synapse load value storage circuit 101 includes a capacitor CA which stores the synapse load value information in the form of charges. The amount of charges stored in the capacitor CA is so continuously changeable that the synapse load value stored in the synapse load value storage circuit 101 is changed in an analog fashion.
The learning control circuit 110 includes a terminal P which receives a control signal Acp indicating execution/non-execution of learning, a terminal C which receives a signal C+/- indicating a learning phase (plus or minus phase), an input terminal S1 which receives the state signal Si, another input terminal S2 which receives the other state signal Sj, a terminal Ip which generates a first correction signal I for increasing the synapse load value stored in the synapse load value storage circuit 101 in response to the states of the state signals Si and Sj in the learning mode, and another terminal Dp which generates a second correction signal D for reducing the synapse load value stored in the synapse load value storage circuit 101 in response to the state signals Si and Sj in the learning mode.
The synapse load correction circuit 103 includes a terminal V which receives the first correction signal I, a terminal L which receives the second correction signal D, and an output terminal M which generates a signal for adjusting the synapse load value stored in the synapse load value storage circuit 101 in response to the correction signals I and D received in the terminals V and L.
The capacitor CA included in the synapse load value storage circuit 101 has an electrode connected to a node N and another electrode coupled to receive a reference potential V. The amount of charges stored in the capacitor CA is increased or decreased in response to the signal received from the synapse load correction circuit 103.
Each of the synapse coupling expressing circuits 105 and 107, which are identical in structure to each other, includes a state signal input terminal Vs for receiving the state signal S (Si or Sj), a terminal Vc for receiving the synapse load value information stored in the synapse load value storage circuit 101, and a terminal Io for outputting a current (loaded current) indicating the product of the state signal S and the synapse load value.
FIG. 14 illustrates an exemplary structure of the learning control circuit 110 shown in FIG. 13. Referring to FIG. 14, the learning control circuit 110 includes an inverter circuit G2 which receives the learning phase indication signal C.+-., a NAND circuit G3 which receives the state signals Si and Sj supplied to the input terminals S1 and S2, a NOR circuit G4 which receives the learning control signal Acp supplied to the terminal P and outputs from the inverter circuit G2 and the NAND circuit G3, and another NOR circuit G5 which receives the control signal Acp, the learning phase indication signal C.+-. and an output from the NAND circuit G3. The NOR circuit G4 generates the first correction signal I, while the NOR circuit G5 generates the second correction signal D. The operation of this learning control circuit 110 is now described.
In non-learning, the control signal Acp is fixed at a high level. In this case, both outputs of the NOR circuits G4 and G5 are fixed at low levels regardless of the logical states of the state signals Si and Sj received at the terminals S1 and S2, respectively, and no correction signals I and D are generated. Namely, the synapse load value stored in the synapse load value storage circuit 101 is not changed in this case.
In learning, on the other hand, a pulse signal having a constant period and a constant pulse width is supplied to the terminal P. In this case, the correction signals generated from the terminals Dp and Ip are changed in response to the learning phase indication signal C+ received in the learning control circuit C. In the plus phase, the learning phase indication signal C.+-. is set at a high level, so that the output D of the NOR circuit G5 is fixed at a low level. The first correction signal I which is transmitted from the NOR circuit G4 to the terminal Ip is an inverted signal of the pulse signal Acp, since the output of the NAND circuit G3 goes low only when both state signals Si and Sj are at high levels. The synapse load correction circuit 103 increases the synapse load value stored in the synapse load value storage circuit 101 in response to the number of pulses of the first correction signal I (.DELTA.Wji&gt;0).
In the minus phase, on the other hand, the learning phase indication signal C.+-. is set at a low level, so that the output signal I of the NOR circuit G4 is fixed at a low level. The NOR circuit G5 functions as an inverter circuit since the output of the NAND circuit G3 goes low only when both state signals Si and Sj are at high levels, to generate an inverted signal of the pulse signal Acp as the output signal (second correction signal) D. The synapse load correction circuit 103 reduces the synapse load value stored in the synapse load value storage circuit 101 in response to the second correction signal D (.DELTA.Wij&lt;0). Namely, the learning control circuit 110 implements the following learning rules: EQU .DELTA.W.sup.+ ji=.eta..multidot.Si.multidot.Sj EQU .DELTA.W.sup.- ji=-.eta..multidot.Si.multidot.Sj
where .eta. corresponds to the number of pulses supplied to the terminal P.
The signs provided on the amount .DELTA.W of correction and the coefficient .eta. correspond to the learning phases.
The control signal Acp and the learning phase indication signal C.+-. received at the terminals P and C are supplied by an external control circuit (not shown).
FIG. 15 illustrates the structures of the synapse coupling expressing circuit 107, which generates the dendrite signal Wji.multidot.Si from the first state signal Si in the synapse expressing circuit SY shown in FIG. 13, as well as the associated circuits. The synapse coupling expressing circuit 105 shown in FIG. 13 is similar in structure to this synapse coupling expressing circuit 107. Referring to FIG. 15, the synapse coupling expressing circuit 107 includes p-channel MOS (insulated gate type field effect) transistors PT1 and PT2 forming a first current path circuit, p-channel MOS transistors PT3 and PT4 forming a second current path circuit, and p-channel and n-channel MOS transistors PT5 and NT1 forming a third current path circuit. The p-channel MOS transistor PT5 is complementarily connected with the n-channel MOS transistor NT1 between a reference voltage (e.g., supply potential) node Vdd and a ground potential node VGND, to form an inverter circuit which inverts the state signal Si received at a terminal Vs.
The p-channel MOS transistor PT1 has a source connected to the reference voltage node Vdd, a gate connected to an output node N2 of the synapse load value storage circuit 101, and a drain connected to a source of the p-channel MOS transistor PT2. The p-channel MOS transistor PT2 has a gate connected to an output node N10 of the inverter circuit formed by the transistors PT5 and NT1, and a drain connected to a synapse coupling current output node Io.
The p-channel MOS transistor PT3 has a source connected to the reference voltage node Vdd, a gate connected to a bias voltage supply node Vb, and a drain connected to a source of the p-channel MOS transistor PT4. The p-channel MOS transistor PT4 has a gate connected to the state signal input node Vs, and a drain connected to the synapse coupling current output node Io.
The p-channel MOS transistor PT5 has a source connected to the reference voltage node Vdd, a gate connected to the state signal input node Vs, and a drain which is connected to a source of the n-channel MOS transistor NT1.
The p-channel MOS transistors in each current path circuit have the same gate widths, i.e., the same conductances. However, the gate widths of the p-channel MOS transistors PT1 and PT2 are set at larger values to be doubled, for example, as compared with those of the p-channel MOS transistors PT3 and PT4. Thus, the current path circuit formed by the transistors PT1 and PT2 can feed a larger current than that formed by the transistors PT3 and PT4.
The synapse load value storage circuit 101 is formed by a capacitor C0. This capacitor CO has an electrode connected to the node N2, and another electrode connected to the reference voltage node Vdd through a node N3. The capacitor CO and the node N2 correspond to the capacitor CA and the node N shown in FIG. 13, respectively.
The synapse load correction circuit 103 includes a capacitor C1 which is provided between the input terminal V for receiving the first correction signal I and a node N5, another capacitor C2 which is provided between the input terminal L for receiving the second correction signal D and a node N6, diodes D2 and D1 which are forward-connected between the node N2 and the reference voltage supply node Vdd (node N3), and diodes D4 and D3 which are forward-connected between the bias voltage supply node Vb (node N1) and the node N2.
The capacitor C1 and the diodes D1 and D2 provide a path for extracting positive charges stored in the node N2 of the capacitor CO in response to the first correction signal I which is received at the first correction signal input terminal V. The capacitor C2 and the diodes D3 and D4 provide a path for injecting positive charges into the capacitor CO in response to the second correction signal D received at the second correction signal input terminal L.
In general, the bias voltage Vb and the reference voltage Vdd, which is an operating power source voltage, for example, satisfies the following relation: EQU VGND.ltoreq.Vb&lt;Vdd
The voltage nodes and the voltages transmitted thereto are denoted by the same symbols. The operation is now described.
When the pulsing correction signal I is supplied to the capacitor C1 through the terminal V, positive charges are extracted from the capacitor CO by a charge pumping operation of the capacitor C1, to lower the potential the node N2. Every time the pulse of the second correction signal D is supplied to the capacitor C2, on the other hand, positive charges are injected into the capacitor C0 to increase the potential of the node N2. Due to this structure, the single capacitor C0 expresses excitatory coupling and inhibitory coupling. The operation of the synapse coupling expressing circuit 107 is now described.
(i) When the state signal Si is at a low levels the transistors PT4 and PT5 enter ON states and the transistors PT2 and NT1 enter OFF states. The potential of the node N10 goes to the reference voltage level Vdd through the transistor PT5. Thus, a constant current determined by the bias voltage Vb, which is supplied to the gate of the transistor PT3, flows from the output terminal Io.
(ii) When the state signal Si is at a high level, on the other hand, the transistor PT4 enters an OFF state and the transistors PT2 and NT1 enter ON states. A current Ids which is determined by a gate potential of the transistor PT1 with reference to its source potential, i.e., a charge potential Vc of the capacitor CO (potential of the node N2) flows from the output terminal Io. When an amount Q0 of charges stored in the node N2 of the capacitor C0 is zero, the potential Vc of the node N2 is equal to the reference voltage Vdd. The source potential of the transistor PT1 is at the reference voltage level Vdd. Thus, a current corresponding to a potential (-Vdd+Vc)=0 flows from the reference voltage supply node Vdd to the output terminal Io through the transistors PT1 and PT2.
When the node N2 of the capacitor C0 stores a negative amount -Q0 of charges, the potential Vc of the node N2 is (Vdd-Q0.multidot.Ca) and the gate potential of the transistor PT1 is -Q0.multidot.Ca, whereby the impedance of the p-channel MOS transistor PT1 is reduced and the amount of the flowing current is increased. When the amount of the current exceeds an amount of current which flows through the transistor PT3 when the state signal Si is at a low level, excitatory coupling is expressed. Symbol Ca represents the capacitance of the capacitor C0.
Positive charges are extracted from the capacitor C0 every time the first correction signal I is received, whereby the impedance of the transistor PT1 is reduced and the value of the current Ids flowing to the output terminal Io is increased. On the other hand, positive charges are injected into the capacitor C0 every time the second correction signal D is received, whereby the value of the current Ids supplied from the transistor PT1 to the node N4 is reduced. Thus, it is possible to express both of excitatory coupling and inhibitory coupling by the single capacitor C0, by adjusting the amount of charges stored in the capacitor CO in response to the correction signals I and D in the learning mode. Further, it is also possible to set the synapse load value expressed by the capacitor C0 at an arbitrary value since this synapse load value is represented by the amount of charges stored in the capacitor C0. Description is now made on an operation of adjusting the amount of charges stored in the capacitor C0 in the learning mode thereby correcting the synapse load value stored in the synapse load value storage circuit 101.
When an electrode of the capacitor C0 connected to the node N2 stores a negative amount -Q0 of charges in the synapse load value storage circuit 101 formed by the capacitor C0, the following voltage is developed at the node N2 which is connected to the gate of the p-channel MOS transistor PT1: EQU Vc=(Vdd-Q0.multidot.Ca)
where Ca represents the capacitance of the capacitor CO. Therefore, the voltage Vc of the node N2 is equal to Vdd when Q0=0, and reduced as the value Q0 is increased.
In the current path circuit including the p-channel MOS transistor PT1, a gate-to-source voltage (hereinafter simply referred to as "gate voltage") of the p-channel MOS transistor PT1 is -(Vdd-Vc). A constant current defined by this gate voltage -(Vdd-Vc) flows to the node N4 through the transistor PT1. When Q0=0, therefore, no current flows through the transistor PT1 since Ids=0, while the potential of the node N2 is reduced and the current Ids flowing through the transistor PT1 is increased as the amount Q0 of charges is increased. The operation of the synapse load correction circuit 103, which is formed by two charge pumping circuits, is now described with reference to FIGS. 16A and 16B showing the respective charge pumping circuits.
With reference to FIG. 16A, an operation for injecting positive charges into the node N2 of the capacitor C0 is now described. Referring to FIG. 16(A), the pulse signal D is supplied to a circuit which is formed by diodes D13 and D14 and a capacitor C12, causing a charge pumping operation for injecting positive charges into a capacitor C20. The diode D13 has an anode connected to a node N26, and a cathode connected to one electrode of the capacitor C20 through a node N22. The diode D14 has a cathode connected to the node N26, and an anode connected to the bias voltage Vb through a node N21.
The capacitor C12 has an electrode connected to the node N26, and another electrode connected to receive the pulse signal D through a node N28. The operation of the circuit shown in FIG. 16(A) is now described.
Consider that the pulse signal D is supplied to the node N28. When the pulse signal D falls from a high level to a low level, the potential of the node N26 falls in a negative direction due to a capacitive coupling of the capacitor C12, whereby the diode D14 enters an ON state. Thus, a current i1 flows from the node N21 to the node N26. At this time, the diode D13 is in an OFF state.
When the pulse signal D rises from a low level to a high level, the potential of the node N26 is increased by a charge pumping operation through the capacitor C12, whereby the diode D13 enters an ON state and the diode D14 enters an OFF state. Thus, a current i2 flows from the node N26 to the node N22. The values of the currents i1 and i2 are determined by the capacitance of the capacitor C12, an amount Q20 of charges stored in the capacitor C20, forward I-V (current-voltage) characteristics of the diodes D13 and D14, and the pulse width of the pulse signal D. Namely, the current flows into the node N22 every cycle of the pulse signal D to charge the capacitor C20, thereby increasing the amount of charges (positive charges) stored in the capacitor C20. The potential of the node N22 is increased by application of the pulse signal D, which corresponds to the second correction signal D.
With reference to FIG. 16B, a charge pumping operation for extracting positive charges from the capacitor CO is now described. Referring to FIG. 16B, the charge pumping operation is carried out by diodes D11 and D12 and a capacitor C11. The diode D11 has a cathode connected to the reference voltage node Vdd through a node N13, and an anode connected to a node N15. The diode N12 has a cathode connected to the node N15, and an anode connected to one electrode of a capacitor C10 through a node N12. The capacitor C11 has an electrode connected to the node N15, and another electrode connected to receive the pulse signal (first correction signal) I through a node N17. The other electrode of the capacitor C10 is connected to the reference voltage node Vdd through the node N13. The operation is now described.
The pulse signal I is supplied to the node N17. When the pulse signal I falls from a high level to a low level, the potential of the node N15 falls by capacitive coupling of the capacitor C11, whereby the diode D12 enters an 0N state and the diode D11 enters an OFF state so that a current i3 flows from the node N12 to the node N15.
When the pulse signal I rises from a low level to a high level, on the other hand, the potential of the node N15 is increased by a charge pumping operation of the capacitor C11, so that the diode D11 enters an ON state when the potential of the node N15 exceeds the reference voltage Vdd. The diode D12 is in an OFF state, and a current i4 flows from the node N15 to the reference voltage node Vdd (node N13) through the node N13. This current i4 is supplied from the capacitor C10, and a current flows from the node N12 to the reference voltage node Vdd through the node N15 every cycle of the pulse signal I, to reduce the amount of positive charges stored in the capacitor C10. The amounts of the currents i3 and i4 are determined by the capacitances of the capacitors C10 and C11, the amount of charges stored in the capacitor C10, forward I-V characteristics of the diodes D11 and D12, and the pulse width of the pulse signal I. The pulse signal I is used as the first correction signal I, so that the amount of charges stored in the capacitor C10 can be adjusted according to the pulse number thereof.
The synapse load correction circuit shown in FIG. 15 is obtained by connecting the capacitors C20 and C10 of the two charge pumping circuits shown in FIGS. 16A and 16B in common. Namely, the nodes N22 and N12 shown in FIGS. 16A and 16B form a common node, while the capacitors C10 and C20 form a common capacitor.
Relations between the elements shown in FIG. 15 and those shown in FIGS. 16A and 16B are as follows: A common capacitor formed by the capacitor C20 (FIG. 16A) and the capacitor C10 (FIG. 16B) corresponds to the capacitor C0 (FIG. 15), and the diodes D11, D12, D13 and D14 (FIGS. 16A and 16B) correspond to the diodes D1, D2, D3 and D4 shown in FIG. 15, respectively. The capacitors C12 and C11 shown in FIGS. 16A and 16B correspond to the capacitors C2 and C1 shown in FIG. 15, respectively. The nodes N28 and N17 shown in FIGS. 16A and 16B correspond to the nodes L and V shown in FIG. 15, respectively. When a pulse signal is supplied to the node V, the amount of charges (negative charges) stored in the capacitor C0 is increased by the diodes D1 and D2 and the capacitor C1, while the amount of charges (negative charges) stored in the capacitor C0 is reduced when the pulse signal is supplied to the node L.
Due to the aforementioned structure, it is possible to control increase and decrease of the amount of charges (negative charges) stored in the capacitor C0 by the pulse numbers or pulse widths of the pulse signals, i.e., the first and second correction signals I and D, which are supplied to the nodes V and L, respectively. In other words, it is possible to control the voltage Vc of the node N2 which determines the amount of the current flowing out from the output terminal Io by the pulse signals supplied to the synapse load correction circuit 103.
In the aforementioned synapse expressing circuit, it is possible to easily correct the synapse load value with the pulse signals, while this synapse load value, being determined by the amount of charges stored in the capacitor, is analogously changed, so that it is possible to implement an arbitrary synapse load value. Thus, it is possible to obtain a synapse expressing circuit having a learning function with a small number of elements, thereby implementing a highly integrated neural network semiconductor chip having a learning function. In such a synapse expressing circuit, however, the following problem is caused since the synapse load value is expressed by the amount of charges stored in the capacitor.
FIG. 17 illustrates the structure of the capacitor C0 forming the synapse load value storage circuit 101. Referring to FIG. 17, the capacitor C0 for expressing a synapse load value comprises a first conductive layer 203 which is serves as an electrode and is formed on a semiconductor substrate 205 with an underlying insulating film 204, and a second conductive layer 201 which is formed on the first conductive layer 203 with an insulating layer 202 as a dielectrics underlaid. In such a structure, capacitance of the capacitor C0 is at a constant value which is determined by the thickness of the insulating layer 202 and the facing area of the conductive layers 201 and 203. While the first and second conductive layers 203 and 201 may be connected to any nodes, the second conductive layer 201 is connected to the node N3 and the first conductive layer 203 is connected to the node N1 in FIG. 17, for example.
The synapse load value is provided by the amount of charges stored in the node N2. This synapse load value must be maintained at a constant value during an associative or normal operation. Due to the inherent property of the capacitor C0, the stored charges leak to change the synapse load value, and hence it is impossible to carry out a correct associative or normal operation. The path of such a charge leakage is now described.
FIG. 18 illustrates a connection of diodes included in the synapse load correction circuit. Referring to FIG. 18, diodes D801 and D802 correspond to the diodes D1 and D2 or D3 and D4 shown in FIG. 15. The diode D801 is forward-connected between nodes b and a, while the diodes D802 is forward-connected between nodes c and b. Charges are injected or extracted through the node b in response to a pulse signal (correction signal).
As shown in FIG. 19, a series body of the diodes D801 and D802 is expressed by a single p-channel MOS transistor PT800. Referring to FIG. 19, the p-channel MOS transistor PT800 has a substrate region connected to the node a, an electrode and a gate electrode connected together to the node b, and another conduction terminal connected to the node c.
FIG. 20 illustrates the sectional structure of the p-channel MOS transistor PT800 shown in FIG. 19. Referring to FIG. 20, the p-channel MOS transistor PT800 is formed in an N-type well 851 on the surface of a p-type semiconductor substrate 850. The p-channel MOS transistor PT800 includes a high impurity concentration N.sup.+ region 852 which is connected to the node a, a high impurity concentration p.sup.+ region 853 which is connected to the node b, and a high impurity concentration p.sup.+ region 854 which is connected to the node c. A gate electrode 855 is formed on a channel region between the P.sup.+ regions 853 and 854 with an insulating film (gate insulating film) 856 thereunder. This gate electrode 855 is connected to the node a through the N.sup.+ region 852.
The diode D801 is formed by the P.sup.+ region 853 and the N.sup.+ region 852. The diode D802 is formed by the P.sup.+ region 853, a channel region, i.e., a surface region of the N-type well 851 under the gate electrode 855, and the P.sup.+ region 854.
In the structure shown in FIG. 20, still another diode is formed through the node a, the N.sup.+ region 852, the N-type well 851 and the P.sup.+ region 854. This diode, which is formed between the nodes a and c, exerts no significant influence on any charge pumping operation. In the element structure shown in FIG. 20, the diode D801 is expressed by a P-N junction diode, whereby it is possible to reduce a reversely flowing discharge current, i.e., a leakage current flowing from the node a to the node c, as well as to reduce parasitic capacitance associated with the node b.
When the element structure shown in FIG. 20 is used to form a series body of diodes, depletion layers 861 and 862 are caused in P-N junction portions. Charges are recombined in or diffused through such depletion layers 861 and 862. In the structure shown in FIG. 20, for example, the node c is connected to the electrode providing the synapse load value of the capacitor C0 of the synapse load value storage circuit 101, i.e., the node N2, or the node N1 providing the bias voltage Vb. The node a is connected to the reference voltage node Vdd or the node N2 in the structure shown in FIG. 15. Thus, the amount of charges stored in the capacitor C0 (see FIG. 15) is changed by recombination or diffusion of charges in the depletion layers 861 and 862, and hence the synapse load value cannot be maintained constant.
Still another leakage path Ip of charges is formed through a depletion layer or an inversion layer which is formed under an element isolating film 870 for electrically isolating adjacent elements from each other.
When the node a or c is connected to the constant voltage Vdd or Vb, this voltage may be changed in an operation to generate hole-electron pairs in the N-type well region 851 by impact ionization or the like. The generated charges may change the amount of charges stored in the capacitor C0. When the amount of charges stored in the capacitor C0 is changed by such a phenomenon, the synapse load value obtained by a learning cannot be maintained constant during the associative or normal operations.
FIG. 21 shows a MOS capacitive element, which may be employed as the capacitor C0. In this case, an electrode of the capacitive element is provided by a high impurity concentration N.sup.+ region 891 which is formed on the surface of a semiconductor substrate 890, and another electrode 893 is formed on this electrode 891 with an insulating film 892 thereunder. Since the electrode 891 is formed by the diffusion region of high impurity concentration, charges inevitably leak from this diffusion region to the semiconductor substrate 890, to change the synapse load value with a lapse of time.
In other words, it is impossible to implement a neural network which can stably hold respective synapse load values being set by learning or the like for a long time to provide reliable operations.